Repository navigation
Support test automation exclusion annotations in JUnit4 - #12940
calvinbayer wants to merge 8 commits into
Conversation
CI Visibility Test Environment - sbt-scalatestJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
CI Visibility Test Environment - nebula-release-pluginJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
This comment has been minimized.
This comment has been minimized.
CI Visibility Test Environment - netflix-zuulJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
CI Visibility Test Environment - reactive-streams-jvmJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
CI Visibility Test Environment - pass4sJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
CI Visibility Test Environment - heliboardJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
CI Visibility Test Environment - sonar-kotlinJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
CI Visibility Test Environment - jolokiaJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
CI Visibility Test Environment - okhttpJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
CI Visibility Test Environment - spring_bootJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
CI Visibility Test Environment - sonar-javaJob Status: 🟢 success
Baseline: median of Tests run to verify that CI Visibility behavior has not regressed in the current PR. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
Protected JUnit4 tests forced to run by TIA’s unskippable category can lose their forced-run tag and metric because reporting ignores the new exclusions.
🤖 Bits Code Review · Commit 193ac29
8939084 to
8028762
Compare
|
@autotest Review |
There was a problem hiding this comment.
More details
Source and test inspection support that protected JUnit4 tests retain blocking failures and backend assignment tags while preserving framework-owned skips and retries.
🤖 Bits Code Review · Commit 8028762
|
@ygree Yes. With The policy selection keeps these controls independent. Your naming suggestion makes that distinction clearer; I’ll leave the API rename open for discussion. |
daniel-mohedano
left a comment
There was a problem hiding this comment.
left some comments and nits, but overall it's looking good, thanks for taking care of these changes 👍
|
@autotest Review |
There was a problem hiding this comment.
More details
Source inspection supports the independent retry and quarantine/disable exclusions while preserving backend assignment tags and TIA selection. Runtime behavior was not exercised.
🤖 Bits Code Review · Commit 259ca51
There was a problem hiding this comment.
More details
Source review supports independent retry and failure-protection exclusions while retaining backend assignment tags. The completed static reviewer reported no actionable defects, and human PR discussion raised no additional unresolved correctness concern.
🤖 Bits Code Review · Commit 259ca51
259ca51 to
be2136e
Compare
|
@autotest Review |
There was a problem hiding this comment.
More details
The exclusion controls remain independent: preventing failure suppression leaves retries available, while disabling retries preserves quarantine and disabling. The human PR discussion confirms this behavior is intentional; static review found no actionable defect.
🤖 Bits Code Review · Commit be2136e
be2136e to
143fa8f
Compare
|
@autotest Review |
There was a problem hiding this comment.
Parameterized JUnit4 display names containing nested brackets can silently bypass @NoAutoRetry and omit its exclusion telemetry because method lookup fails.
🤖 Bits Code Review · Commit 143fa8f
|
@autotest Review |
There was a problem hiding this comment.
More details
Source inspection supports independent retry and failure-suppression exclusions across JUnit4 skip decisions, execution policies, and reporting, including parameterized method resolution. Runtime behavior was not exercised.
🤖 Bits Code Review · Commit 855c0da
Shepherd verification — JUnit4 native tests and packaged Java smokeTested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)
The regression reproduces the annotation lookup failure for The packaged smoke used the normal Crook workflow with Scope: Jolokia provides a packaged Java/TestNG smoke; the native JUnit4 instrumentation tests prove the JUnit4 annotation behavior. The configured HeliBoard JUnit4 playground could not complete normal setup because the Android SDK was absent. No SDK/NDK installation or dependency-skipping workaround was used. This verification used local Mockdog; it did not validate the Datadog UI or production backend. Normal Crook cleanup restored the playground dependency state, removed the injected agent, and stopped its owned Mockdog process. The source worktree remained clean. |
Resolve reflected methods before the first parameter bracket without changing reported test names or parameter JSON. Native regression proves that bracketed labels keep NoAutoRetry declarations and newline labels retain their behavior, with independent body counters and an active ATR control.
855c0da to
5a0abf5
Compare
What Does This Do
Add public
@NoAutoRetryand@PreventFailureSuppressionannotations and JUnit4 support. The exclusions independently control Datadog retries and quarantine/disabling while preserving framework-owned skips, explicit retries, Test Impact Analysis selection, and backend-assigned management tags.Motivation
Let developers protect individual critical tests from Datadog automation. Each annotation also exposes a
TAGconstant for the native tag/group integrations in the later framework PRs.Additional Notes
First PR in the Java framework stack. Shared declaration resolution, execution policy, reporting, and public annotation Javadocs live here. The declaration resolver uses cached annotation types and native tag constants; JUnit4 supplies the same source and category metadata to skip decisions and reporting.
Validation: focused declaration-resolver, public API, execution-policy and event-handler tests; the instrumented JUnit4 suite; formatting checks; and public API Javadoc generation. The regressions verify independent flags, disabled/quarantined/attempt-to-fix assignments, blocking protected failures, annotation telemetry, and TIA forced-run reporting.